Motion and gesture input from a wearable device

ABSTRACT

This relates to a device that detects a user&#39;s motion and gesture input through the movement of one or more of the user&#39;s hand, arm, wrist, and fingers, for example, to provide commands to the device or to other devices. The device can be attached to, resting on, or touching the user&#39;s wrist, ankle or other body part. One or more optical sensors, inertial sensors, mechanical contact sensors, and myoelectric sensors can detect movements of the user&#39;s body. Based on the detected movements, a user gesture can be determined. The device can interpret the gesture as an input command, and the device can perform an operation based on the input command. By detecting movements of the user&#39;s body and associating the movements with input commands, the device can receive user input commands through another means in addition to, or instead of, voice and touch input, for example.

FIELD

This relates generally to a device that detects a user's motion andgesture input to provide commands to the device or to other devices. Inparticular, the device can use one or more sensors to determine a user'smotion and gesture input based on movements of the user's hand, arm,wrist, and fingers.

BACKGROUND

Many existing portable electronic devices use voice or touch input as amethod for the user to communicate commands to the devices or to controlthe devices. One example is a voice command system, which can mapspecific verbal commands to operations, for example, to initiate dialingof a telephone number by speaking the person's name. Another example isa touch input system, where the user can choose a specific devicesetting, such as adjusting the volume of the speakers, by touching aseries of virtual buttons or performing a touch gesture. While voice andtouch input can be an effective way to control a device, there may besituations where the user's ability to speak the verbal command orperform the touch gesture may be limited.

SUMMARY

This relates to a device that detects a user's motion and gesture inputthrough the movement of one or more of the user's hand, arm, wrist, andfingers, for example, to provide commands to the device or to otherdevices. The device can be attached to, resting on, or touching theuser's wrist, ankle or other body part. One or more optical sensors,inertial sensors, mechanical contact sensors, and myoelectric sensors,to name just a few examples, can detect movements of the user's body.Based on the detected movements, a user gesture can be determined. Thedevice can interpret the gesture as an input command, and the device canperform an operation based on the input command. By detecting movementsof the user's body and associating the movements with input commands,the device can receive user input commands through another means inaddition to, or instead of, voice and touch input, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C illustrate systems in which examples of the disclosure canbe implemented.

FIG. 2A illustrates an exemplary depiction of a human hand according toexamples of the disclosure.

FIG. 2B illustrates a cross-sectional view of a human wrist according toexamples of the disclosure.

FIGS. 3A-3H illustrate exemplary finger and wrist movements according toexamples of the disclosure.

FIG. 4 illustrates an exemplary configuration of a wearable deviceattached to the wrist of a user according to examples of the disclosure.

FIG. 5A illustrates a cross-sectional view of a wrist and an exemplarydevice with motion and gesture sensing using optical sensors accordingto examples of the disclosure.

FIG. 5B illustrates a top view of a wrist and an exemplary device withmotion and gesture sensing using optical sensors according to examplesof the disclosure.

FIG. 6 illustrates a plan view of an exemplary device with motion andgesture sensing using inertial sensors according to examples of thedisclosure.

FIG. 7A illustrates a cross-sectional view of a wrist and an exemplarydevice with motion and gesture sensing using mechanical contact sensorsaccording to examples of the disclosure.

FIG. 7B illustrates a cross-sectional view of a wrist and an exemplarydevice with motion and gesture sensing using optical sensors located inthe strap according to examples of the disclosure.

FIG. 7C illustrates a close-up view of the strap according to examplesof the disclosure.

FIG. 8 illustrates a cross-sectional view of a wrist and an exemplarydevice with motion and gesture sensing using myoelectric sensorsaccording to examples of the disclosure.

FIG. 9A illustrates exemplary gestures and corresponding commandsaccording to examples of the disclosure.

FIG. 9B illustrates an exemplary process flow for determining a commandbased on the user's movement according to examples of the disclosure.

FIG. 9C illustrates an exemplary process flow for recording user-definedgestures according to examples of the disclosure.

FIGS. 9D-9E illustrate exemplary hand and wrist movement according toexamples of the disclosure.

FIGS. 9F-9H illustrate exemplary finger movements associated with signlanguage according to examples of the disclosure.

FIG. 10 illustrates an exemplary block diagram of a computing systemcomprising one or more motion and gesture sensors for determining auser's gesture or motion according to examples of the disclosure.

FIG. 11 illustrates an exemplary configuration in which a device isconnected to a host according to examples of the disclosure.

DETAILED DESCRIPTION

In the following description of examples, reference is made to theaccompanying drawings in which it is shown by way of illustrationspecific examples that can be practiced. It is to be understood thatother examples can be used and structural changes can be made withoutdeparting from the scope of the various examples.

Various techniques and process flow steps will be described in detailwith reference to examples as illustrated in the accompanying drawings.In the following description, numerous specific details are set forth inorder to provide a thorough understanding of one or more aspects and/orfeatures described or referenced herein. It will be apparent, however,to one skilled in the art, that one or more aspects and/or featuresdescribed or referenced herein may be practiced without some or all ofthese specific details. In other instances, well-known process stepsand/or structures have not been described in detail in order to notobscure some of the aspects and/or features described or referencedherein.

Further, although process steps or method steps can be described in asequential order, such processes and methods can be configured to workin any suitable order. In other words, any sequence or order of stepsthat can be described in the disclosure does not, in and of itself,indicate a requirement that the steps be performed in that order.Further, some steps may be performed simultaneously despite beingdescribed or implied as occurring non-simultaneously (e.g., because onestep is described after the other step). Moreover, the illustration of aprocess by its depiction in a drawing does not imply that theillustrated process is exclusive of other variations and modificationthereto, does not imply that the illustrated process or any of its stepsare necessary to one or more of the examples, and does not imply thatthe illustrated process is preferred.

This disclosure relates to a device that detects a user's motion andgesture input to provide commands to the device or to other devices. Thedevice can be attached to, resting on, or touching a user's wrist, ankleor other body part. One or more optical sensors, inertial sensors,mechanical contact sensors, and myoelectric sensors, to name just a fewexamples, can allow the device to detect movements of a user's body,such as the user's hand, arm, wrist, and fingers. Based on the detectedmovements, a user gesture can be determined. The device can interpretthe gesture as an input command, and the device can perform an operationbased on the input command. By detecting movements of the user's bodyand associating the movements with input commands, the device canreceive user input commands through another means in addition to, orinstead of, voice and touch input, for example.

In some examples, optical sensing can employ light sources and lightsensors located on the device itself or located in the strap attached tothe device. The light sources and light sensors can generate areflectance profile from the reflectance of the light off the user'stendons, skin, muscles, and bones. In some examples, inertial sensingcan employ an accelerometer and gyroscope to determine rigid bodymotions based on the change in motion along the axes and the change inorientation of the device attached to, resting on, or touching theuser's hand, ankle or other body part. In some examples, mechanicalcontact sensing can be employed by using at least one flexible materialaround the user's body part, such as the wrist, that conforms to theuser's movement. In some examples, myoelectric sensors can allow thedevice to detect the electrical signal or the change in capacitance inthe tendons coupled with the user's movement.

Representative applications of methods and apparatus according to thepresent disclosure are described in this section. These examples arebeing provided solely to add context and aid in the understanding of thedescribed examples. It will thus be apparent to one skilled in the artthat the described examples may be practiced without some or all of thespecific details. Other applications are possible, such that thefollowing examples should not be taken as limiting.

FIGS. 1A-1C illustrate systems in which examples of the disclosure canbe implemented. FIG. 1A illustrates an exemplary mobile telephone 136that can include a touch screen 124. FIG. 1B illustrates an exemplarymedia player 140 that can include a touch screen 126. FIG. 1Cillustrates an exemplary wearable device 144 that can include a touchscreen 128 and can attach to a user using a strap 146. The systems ofFIGS. 1A-1C be configured for optical sensing, inertial sensing,mechanical contacting sensing, myoelectric sensing, or a combination oftwo or more to determine a user's motion and gesture, as will bedisclosed.

FIG. 2A illustrates an exemplary depiction of a human hand, and FIG. 2Billustrates a cross-sectional view of a human wrist. It should be notedthat although examples of the disclosure may be provided primarily withrespect to a device attached to a user's wrist, and may primarilyillustrate motions of the user's fingers, hand, or arm, other body partssuch as ankles, knees, or the head (to name just a few examples) andtheir associated movements are also contemplated and fall within thescope of this disclosure. Hand 204 can include a plurality of fingers202, a wrist 220, a plurality of tendons 210 and 212, a plurality ofmuscles 230, and a plurality of bones 240. Tendons 210 can be located onthe palm-side of the hand, also known as the palmar side. Tendons 210can also be referred to as flexor tendons. Tendons 212 can be located ona front side of the hand, also known as the dorsal side. Tendons 212 canalso be referred to as extensor tendons. The hand muscles 230 areattached to bones 240 through the plurality of tendons 210 and 212. Whena human moves a muscle or a bone, the human brain sends an electricalsignal through the nervous system to the corresponding nerve. The nervestimulates the muscle with the electrical signal, causing the muscle tocontract or move. Muscle movement can lead to bone movement through theattached one or more tendons.

FIGS. 3A-3H illustrate exemplary finger and wrist movements. FIG. 3Aillustrates abduction of the fingers 302 that can involve abductormuscles. As used herein, the term “abductors” generally refers tomuscles that cause movement away from the center line 350. FIG. 3Billustrates adduction of the fingers 302 that can involve adductormuscles. As used herein, the term “adductors” generally refers tomuscles that cause movement towards the center line 350.

Each finger (except the thumb) can include three joints: themetacarpophalangeal (MCP) joint, proximal interphalangeal (PIP) joint,and distal interphalangeal (DIP) joint. The MCP joints, also known asthe knuckles, are located between the hand and fingers. The PIP jointsare the next set of joints toward the fingernail, and the DIP joints arethe farthest joints of the finger. Abduction of the fingers 302, asillustrated in FIG. 3A, and adduction of the fingers 302, as illustratedin FIG. 3B, can involve moving the MCP joint.

FIGS. 3C-3D illustrate flexion and extension of one or more fingers 302.Flexions 352 and 356 can involve muscles and tendons that bend one ormore fingers 302 towards wrist 320. Flexion 352 can involve the MCPjoint, whereas flexion 356 can involve PIP and DIP joints. Extensions354 and 358 can involve muscles and tendons that cause movement of oneor more fingers 302 away from wrist 320 and can involve all three MCP,PIP, and DIP joints. In some examples, finger flexion can includeflexion of the thumbs, which can result in the user making a fist, forexample.

FIG. 3E illustrates wrist abduction, and FIG. 3F illustrates wristadduction where the user can cause the wrist to move to one of thesides. Wrist abduction or radial deviation, as shown in FIG. 3E, caninvolve movement of the thumb side of the hand towards the radial sideof the forearm. Wrist adduction or ulnar deviation, as shown in FIG. 3F,can involve movement of the little finger side of the hand towards theulnar side of the forearm. FIG. 3G illustrates wrist extension, and FIG.3H illustrates wrist flexion. Wrist extension can involve movement ofthe palm of the hand toward the dorsal side of the forearm. Wristflexion can involve movement of the palm of the hand toward the insideof the forearm.

FIG. 4 illustrates an exemplary configuration of a wearable deviceattached to a wrist according to examples of the disclosure. A user'sarm can include fingers 402, hand 404, and wrist 420. Device 400 can beattached to, resting on, or touching a user's skin at any body part,such as the user's wrist 420. Muscles 430 can be attached to the bonesin fingers 402 through tendons 410. When a user wants to perform any oneof the movements illustrated in FIGS. 3A-3H, fingers 402, wrist 420, andhand 404 can move when the user's brain sends electrical signals tostimulate muscles 430. Muscles 430 can contract in response to thereceived electrical signals. In response to the received electricalsignals, tendons 410, attached to muscles 430, can also contract or moveand can cause move fingers 402, wrist 420, and hand 404 to move. As thetendons contract or move, device 400 can detect the movement of thetendons, the electrical signal, or both. Based on either the tendonmovement or electrical signal or both, device 400 can determine theuser's motion and gesture. The motion and gesture can be interpreted ascommands to the device or another device. In some examples, a hostdevice can perform the determination of the user's motion and gesture,as will be described below.

FIG. 5A illustrates a cross-sectional view of a wrist and an exemplarydevice with motion and gesture sensing using optical sensors accordingto examples of the disclosure. Device 500 can attach to wrist 520 usingstrap 546. In some examples, device 500, strap 546, or both can touchthe skin of wrist 520. Wrist 520 can include tendons 510 and 512 andmuscles 530. Device 500 can include one or more light sources 502 andone or more light sensors 504. Light sources 502 can be directed at theskin, tendons 510, and muscles 530. Light emitted from light sources 502can reflect off the skin, tendons 510, and muscles 530 to create areflectance profile detected by the light sensors 504. The reflectanceprofile can change with movements in the tendons of the flexor/extensorand abductors/adductors muscles. From the reflectance profile, thedevice can determine and distinguish the motions and gestures (e.g.,which finger is moved, how the wrist is bent, etc.). For example,opening a hand or making a fist can cause movement of all tendons,whereas moving an individual finger can cause movement of a singletendon. In some examples, strap 546 can include one or more lightsources 506 and one or more light sensors 508. Light sources 506 can bedirected at and can reflect off the skin, tendons 512, and muscles 530to create another reflectance profile detected by light sensors 508.Although FIG. 5A illustrates four light sources and four light sensors,examples of the disclosure can include any number of light sources andany number of light sensors.

In some examples, one or more light sources 502 and 506 and one or morelight sensors 504 and 508 can have different emission and detectionwavelengths. By emitting and detecting light at different wavelengths, avariety of information can be determined. Device 500 can include opticalsensing at longer wavelengths (e.g., infrared light), shorterwavelengths (e.g., blue or green light), or both. Longer wavelengths canpenetrate deep into the human skin. The longer wavelength light canundergo minimal scattering and absorption and can reflect off of theinternal layers of the human body. For example, an infrared sourceemitting at 950 nm can penetrate 1-2 mm deep into the human skin.Shorter wavelengths may not be able to penetrate as deep as longerwavelengths. For example, deep blue light can reflect off the surface ofsuperficial layers without penetrating into the skin. Green light canpenetrate deeper than blue light to reach blood vessels. Green light canbe absorbed by hemoglobin and can have low back-reflection from theskin. For example, a blue light source can emit at 450 nm and a greenlight source can emit at 550 nm, penetrating 0.1-0.2 mm in depth.

Device 500 can be configured for multi-wavelength illumination andsensing to generate both a spatial and temporal reflectance profilesensitive to changes in the user's skin, tendons, muscles, and bloodvolume as the user moves their wrist and fingers. With the spatial andtemporal reflectance, the device can determine the gesture-inducedinternal structural changes unique to the user.

In some examples, configuring device 500 for multi-wavelength opticalsensing can reduce or eliminate motion artifacts. One or morewavelengths (such as short wavelengths) can detect non-internal changesin the skin, and one or more wavelengths (such as long wavelengths) candetect internal changes. Motion artifacts (i.e., non-internal changes)due to, for example, strap 546 vibrating or moving along wrist 520, canlead to changes in the reflectance of light that reflects mostly off thesurface of the user's skin. Movement of the user's wrist 520 or fingers(i.e., internal changes) can lead to changes in the reflectance of lightthat penetrates into the skin. As a result, a signal measured at shortwavelengths and not at long wavelengths can be associated with motionartifacts and not user movement. The difference between the longwavelength signal and the short wavelength signal can allow the deviceto extract out motion artifacts.

The lights sensors and light sources can be positioned on the device tospecifically measure movement of the tendons or the muscles. FIG. 5Billustrates a top view of a wrist and an exemplary device with motionand gesture sensing using optical sensors according to examples of thedisclosure. Device 500 can attach to a wrist 520 using strap 546. Device500 can include a plurality of light sources 502 and a plurality oflight sensors 504. The plurality of light sources 502 can be positionedsuch that light emitted from the light sources 502 is directed towardsand can reflect off the tendons 510. The plurality of light sensors 504can be positioned near the plurality of lights sources 502 and candetect the reflectance profile. Each one of the tendons 510 can beassociated with a different light source 502 and light sensor 504 pair.When the user flexes or extends the fingers, tendons 510 can cause aripple in the surface of the user's skin located at wrist 520. Each ofthe fingers can cause a ripple at a different location, and the lightsource and light sensor pair can detect the corresponding tendon 510moving closer to or away from the skin surface. As a tendon moves, thegap between the tendon and the light source 502 and light sensor paircan change, resulting in a change in the reflectance profile.

In some examples, light sources 502 and lights sensors 504 can bemulti-functionality sensors where light sources and light sensors can beconfigured to measure other signals. For example, light sources 502 andlight sensors 504 can also be configured as photoplethysmography (PPG)sensors for measuring a user's heart rate or blood pressure.

In some examples, inertial sensors, such as an accelerometer andgyroscope, can detect motions and gestures. FIG. 6 illustrates a planview of an exemplary device with motion and gesture sensing usinginertial sensors according to examples of the disclosure. Device 600 canattach to, rest on, or touch a user's wrist (not shown). Device 600 canalso include an accelerometer and gyroscope 630 to determinetranslational and rotational motion. In some examples, the accelerometerand gyroscope can be separate components. The accelerometer can measurenon-gravitational acceleration and can determine the change in motionalong the x-axis 610, y-axis 612, and z-axis 614. The gyroscope canmeasure the orientation of the device and can determine the pitch 620,roll 622, and yaw 624.

By using an accelerometer, gyroscope, or both to detect rigid bodymotions, the device can determine predefined gestures. Examples of suchmotions can include, but are not limited to, circular wrist motion, handwaving, hand up and down movements, palm up and down movements, and armwaving.

In some examples, one or more light sources such as light sources 502and 506 of FIG. 5, one or more light sensors 504 and 508 of FIG. 5, andan accelerometer and/or a gyroscope such as accelerometer and gyroscope630 of FIG. 6 can be incorporated into a device for optical and inertialsensing. Optical sensing can allow the device to determine wrist andfinger flexion, extension, abduction, and adduction, while the inertialsensing can allow the device to determine translational and rotationalmotion.

In some examples, the device can utilize mechanical contact sensing todetect motions and gestures. FIG. 7A illustrates a cross-sectional viewof a wrist and an exemplary device with motion and gesture sensing usingmechanical sensors according to examples of the disclosure. Device 700can include a strap 746 attached to, resting on, or touching wrist 720.Wrist 720 can include tendons 710 and 712, muscles 730, and bones 740beneath a user's skin. Strap 746 can include a plurality of regions 750,752, 754, and 756. Strap 746 can be made of a flexible material, such asViton. As the user's wrist 720 moves, the strap 746 can stretch orcompress at corresponding regions where the stretching/compressingoccurs. Device 700 can be configured for detecting the stretch in one ormore regions independent of other regions. For example, a user may makea fist gesture. A fist gesture can cause a stretch in strap 746 locatedat regions 752 and 754, while regions 750 and 756 are unaffected (i.e.,no stretching or compressing at regions 750 and 756). In some examples,a plurality of regions can stretch, and the location and intensity ofchange in length or area at the regions can be indicative of a user'sgesture. In some examples, strap 746 can be tightly fitted to wrist 720.

In some examples, strap 746 can be made of a flexible material, and caninclude gauges capable of measuring a change in length or area of theflexible material. For example, one or more strain gauges can attach toor can be located in strap 746. Circuitry included in the device 700 orin the strap 746 can be configured to measure resistance from the one ormore strain gauges. As a region on strap 746 stretches, the resistancecan increase, while a region that compresses can cause a decrease inresistance.

In some examples, strap 746 can have an insufficient amount of frictionforces against wrist 720. As a result of having an insufficient amountof friction forces, strap 746 may slip against the user's skin, leadingto erroneous measurements. FIG. 7B illustrates a cross-sectional view ofa wrist and an exemplary device with motion and gesture sensing usingoptical sensors located in the strap according to examples of thedisclosure. FIG. 7C illustrates a close-up view of the strap accordingto examples of the disclosure. Device 700 can include a strap 760attached to wrist 720. Strap 760 can include an inner band 764 and anouter band 766. Inner band 764 can be made of a flexible material andcan be tightly fitted to wrist 720. Outer band 766 can be made of arigid material. Inner band 764 can include a plurality of opticalfeatures 762, and outer band 766 can include one or more light sources,such as light source 772, and one or more light sensors, such as lightsensor 774. In some examples, strap 760 can include a cosmetic layer768.

Light source 772, located in outer band 776, can emit light towardsoptical features 762, located in inner band 764. The emitted light canreflect off the optical features 762 and can be detected by light sensor774, located in outer band 766. The movement of the user's wrist canlead to movement of the optical features 762, which can cause a changein the reflectance of the light detected by light source 772.

In some examples, the device can include myoelectric sensors to detectmotions and gestures. FIG. 8 illustrates a cross-sectional view of awrist and an exemplary device with motion and gesture sensing usingmyoelectric sensors according to examples of the disclosure. Device 800can include a strap 846 attached to a wrist 820. Wrist 820 can includetendons 810 and 812, muscles 830, and bones 840. Device 800 can includeone or more myoelectric sensors or electrodes 806 and 816. Electrodes806 and 816 can be configured to measure the electrical signal fromtendons 810 and 812. In some examples, electrodes 806 and 816 can beconfigured to measure a capacitance from a body part, such as tendons810 and 812, to the electrodes. A change in capacitance can beassociated with the user movement. The electrical signal or change incapacitance can allow the device to determine a corresponding motion orgesture based on the intensity and location of the electrical signal orthe change in capacitance.

Any one of the optical sensors, inertial sensors, mechanical contactsensors, and myoelectric sensors used individually or together can allowthe device to determine a user's motion, gesture, or both. Hand motionscan include, but are not limited to, wrist movements, opening andclosing of the hand, palm orientated up, down, towards, or away andfinger flexing/extending, and movement of the entire hand in an up,down, left or right direction. One or more hand motions can define agesture input. The device can interpret the gesture input as a command.Exemplary gestures and corresponding commands are illustrated in FIG.9A.

The gestures and associated commands can be pre-defined and stored in adevice database. FIG. 9B illustrates an exemplary process flow fordetermining a command based on the user's movement according to examplesof the disclosure. Process 950 can begin with step 952 where the devicedetects the user's movement. Based on the user's movement, the devicecan determine the gesture (step 954). The device can compare thedetermined gesture to pre-defined gestures located in the database (step956). If there is a match between the user's gesture and a pre-definedgesture, the device can look up the command that is associated with thepre-defined gesture and can perform the command (step 958).

In some examples, the device can include an application programminginterface (API) that can enable applications to record gestures definedby the user and to associate gestures with specific tasks or commands.FIG. 9C illustrates an exemplary process flow for recording user-definedgestures according to examples of the disclosure. In process 960, thedevice can track a gesture or motion history and the task or commandthat typically follows the gesture or motion (step 962). That is, thedevice can learn from past history which commands are associated withwhich gestures. Then, in the future, the device can predict what commandthe user desires to follow a user gesture. When the user moves his orher hand, arm, wrist, or fingers, the device can determine the gesture(step 964). The device can predict the associated command (step 966),and the device can execute the command without direct user interaction(step 968).

For example, a user can begin with their arm and wrist located at theside of their body as illustrated in FIG. 9D. The user may move theirarm and wrist, such that the dorsal side of the wrist is facing up andtowards the user's eye, as illustrated in FIG. 9E. The device candetermine from past history that such a movement occurs when a userwants to look at the display of the device. The device can associate themovement illustrated in FIGS. 9D-9E with the task or command ofautomatically turning on the display and waking up the device. That way,the user no longer has to push a button or tap the display screen towake up the device, and instead, the device can “intelligently” andautomatically wake up the device based on this gesture.

In some examples, the device can include a user-specific calibrationprocedure. The calibration procedure can include an optical calibration,for example, to compensate for anatomic differences between users. Thedevice can display a schematic of fingers, bones, or tendons on thescreen. With the device attached to, resting on, or touching, the user'sbody part, the user can flex or extend each finger. The device candetect each finger or tendon movement and associated information. Theassociated information can be used to establish a baseline. When theuser performs a gesture or movement, the device can compare a signalmeasured from the gesture or movement and can compare the signal to thebaseline.

In addition to detected hand and wrist movements, the device can detectfinger movements. An example application including detecting fingermovements can be detecting sign language. FIGS. 9F-9H illustrateexemplary finger movements associated with sign language according toexamples of the disclosure. As shown in FIG. 9F, a user can sign theletter C, for example, using fingers 902, and device 900 can sense themovement of the tendons located at or near wrist 920. Logic located indevice 900 can determine that the movement of fingers 902 and thegesture illustrated in FIG. 9F corresponds to the user signing theletter C.

In some examples, detecting sign language can include both finger andwrist movements. For example, a user can sign the phrase “Thank You” byextending the fingers 902 and moving the wrist 920 and device 900 awayfrom a user's mouth 990, as illustrated in FIG. 9G. Device 900 candetermine that all fingers 920 are extended by measuring the movement ofthe tendons located at or near wrist 920. Additionally, device 920 candetermine that wrist 920 was moved away from the mouth 990 using theinertial sensors.

In some examples, detecting sign language can include detecting bothfinger and wrist movements in both hands of the user. For example, auser can sign the word “Go” by extending both index fingers 903 and 905for both hands, flexing the remaining fingers 902 and 907 for bothhands, and moving wrists 920 and 921 in an alternating and circularfashion. Devices 900 and 901 can attach to the wrists 920 and 921 todetect the extension of fingers 903 and 905 and the flexion of fingers902 and 907 through the movement of the tendons located at or nearwrists 920 and 921. Devices 900 and 901 can detect the circular movementof the wrists 920 and 921 using the inertial sensors. In some examples,device 901 can send detected gesture and movement signals or informationto device 900 using wired or wireless communications, such as Bluetooth.When device 900 receives the information from device 901, device 900 candetermine that the user is moving both wrists 920 and 921, and fingers902, 903, 905 and 907, and can associate the gestures with acorresponding phrase or command. In some examples, both device 900 and901 can send detected gesture and movement signals or information to ahost device. The host device can process the signals, determine thegesture and movement, and associate the gesture with the correspondingphrase or command. While the figures illustrate device 900 attached tothe user's wrist, examples of the disclosure can include the deviceattached to other body parts.

In some examples, association of the gesture in any of the illustratedabove examples can lead to the task of audibly announcing the associatedphrase or letter through a speaker or displaying the associated phraseor letter on a display, for example. Device 900 can then be, forexample, a sign language interpreter.

FIG. 10 illustrates an exemplary block diagram of a computing systemcomprising one or more motion and gesture sensors for determining auser's gesture or motion according to examples of the disclosure.Computing system 1000 can correspond to any of the computing devicesillustrated in FIGS. 1A-1C. Computing system can include a processor1010 configured to execute instructions to carry out operationsassociated with computing system 1000. For example, using instructionsretrieved from memory, processor 1010 can control the reception andmanipulation of input and output data between components of computingsystem 1000. Processor 1010 can be a single-chip processor or can beimplemented with multiple components.

In some examples, processor 1010 together with an operating system canoperate to execute computer code and produce end user data. The computercode and data can reside within a program storage block 1002 that can beoperatively coupled to processor 1010. Program storage block 1002 cangenerally provide a place to hold data that is being used by computingsystem 1000. Program storage block 1002 can be any non-transitorycomputer-readable storage medium, and can store, for example, historyand/or pattern data relating to gesture and motion values measured byone or more motion and gesture sensors 1004. By way of example, programstorage block 1002 can include Read-Only Memory (ROM) 1018,Random-Access Memory (RAM) 1022, hard disk drive 1008 and/or the like.The computer code and data could reside on a removable storage mediumand be loaded or installed onto the computing system 1000 when needed.Removable storage mediums include, for example, CD-ROM, DVD-ROM,Universal Serial Bus (USB), Secure Digital (SD), Compact Flash (CF),Memory Stick, Multi-Media Card (MMC) and a network component.

Computing system 1000 can also include an input/output (I/O) controller1012 that can be operatively coupled to processor 1010, or it may be aseparate component as shown. I/O controller 1012 can be configured tocontrol interactions with one or more I/O devices. I/O controller 1012can operate by exchanging data between processor 1010 and the I/Odevices that desire to communicate with processor 1010. The I/O devicesand I/O controller 1012 can communicate through a data link. The datalink can be a one way link or a two way link. In some examples, I/Odevices can be connected to I/O controller 1012 through wirelessconnections. By way of example, a data link can correspond to PS/2, USB,Firewire, IR, RF, Bluetooth or the like.

Computing system 1000 can include a display device 1024 that can beoperatively coupled to process 1010. Display device 1024 can be aseparate component (peripheral device) or can be integrated withprocessor 1010 and program storage block 1002 to form a desktop computer(all-in-one machine), a laptop, a handheld, wearable or tablet computingdevice or the like. Display device 1024 can be configured to display agraphical user interfaced (GUI) including perhaps a pointer or cursor aswell as other information. By way of example, display device 1024 can beany type of display including a liquid crystal display (LCD), anelectroluminescent display (ELD), a field emission display (FED), alight emitting diode display (LED), an organic light emitting diodedisplay (OLED) or the like.

Display device 1024 can be coupled to display controller 1026 that canbe coupled to processor 1010. Processor 1010 can send raw data todisplay controller 1026, and display controller 1026 can send signals todisplay device 1024. Data can include voltage levels for a plurality ofpixels in display device 1024 to project an image. In some examples,processor 1010 can be configured to process the raw data.

Computing system 1000 can also include a touch screen 1030 that can beoperatively coupled to processor 1010. Touch screen 1030 can be acombination of sensing device 1032 and display device 1024, where thesensing device 1032 can be a transparent panel that is positioned infront of display device 1024 or integrated with display device 1024. Insome cases, touch screen 1030 can recognize touches and the position andmagnitude of touches on its surface. Touch screen 1030 can report thetouches to processor 1010, and processor 1010 can interpret the touchesin accordance with its programming. For example, processor 1010 canperform tap and event gesture parsing and can initiate a wake of thedevice or powering on one or more components in accordance with aparticular touch.

Touch screen 1030 can be coupled to a touch controller 1040 that canacquire data from touch screen 1030 and can supply the acquired data toprocessor 1010. In some examples, touch controller 1040 can beconfigured to send raw data to processor 1010, and processor 1010 canprocess the raw data. For example, processor 1010 can receive data fromtouch controller 1040 and can determine how to interpret the data. Thedata can include the coordinates of a touch as well as pressure exerted.In some examples, touch controller 1040 can be configured to process rawdata itself. That is, touch controller 1040 can read signals fromsensing points 1034 located on sensing device 1032 and can turn theminto data that the processor 1010 can understand.

Touch controller 1040 can include one or more microcontrollers such asmicrocontroller 1042, each of which can monitor one or more sensingpoints 1034. Microcontroller 1042 can, for example, correspond to anapplication specific integrated circuit (ASIC), which works withfirmware to monitor the signals from sensing device 1032, process themonitored signals, and report this information to processor 1010.

One or both display controller 1026 and touch controller 1040 canperform filtering and/or conversion processes. Filtering processes canbe implemented to reduce a busy data stream to prevent processor 1010from being overloaded with redundant or non-essential data. Theconversion processes can be implemented to adjust the raw data beforesending or reporting them to processor 1010.

In some examples, sensing device 1032 is based on capacitance. When twoelectrically conductive members come close to one another withoutactually touching, their electric fields can interact to form acapacitance. The first electrically conductive member can be one or moreof the sensing points 1034, and the second electrically conductivemember can be an object 1090 such as a finger. As object 1090 approachesthe surface of touch screen 1030, a capacitance can form between object1090 and one or more sensing points 1034 in close proximity to object1090. By detecting changes in capacitance at each of the sensing points1034 and noting the position of sensing points 1034, touch controller1040 can recognize multiple objects, and determine the location,pressure, direction, speed and acceleration of object 1090 as it movesacross touch screen 1030. For example, touch controller 1090 candetermine whether the sensed touch is a finger, tap or an objectcovering the surface.

Sensing device 1032 can be based on self-capacitance or mutualcapacitance. In self-capacitance, each of the sensing points 1034 can beprovided by an individually charged electrode. As object 1090 approachesthe surface of touch screen 1030, the object can capacitively couple tothose electrodes in close proximity to object 1090, thereby stealingcharge away from the electrodes. The amount of charge in each of theelectrodes can be measured by the touch controller 1040 to determine theposition of one or more objects when they touch or hover over the touchscreen 1030. In mutual capacitance, sensing device 1032 can include atwo layer grid of spatially separated lines or wires, although otherconfigurations are possible. The upper layer can include lines in rows,while the lower layer can include lines in columns (e.g., orthogonal).Sensing points 1034 can be provided at the intersections of the rows andcolumns. During operation, the rows can be charged, and the charge cancapacitively couple from the rows to the columns. As object 1090approaches the surface of the touch screen 1030, object 1090 cancapacitively couple to the rows in close proximity to object 1090,thereby reducing the charge coupling between the rows and columns. Theamount of charge in each of the columns can be measured by touchcontroller 1040 to determine the position of multiple objects when theytouch the touch screen 1030.

Computing system 1000 can also include one or more sensors 1004proximate to a wrist of a user. Sensors 1004 can be at any one of theabove disclosed optical sensors, inertial sensors, mechanical contactsensors, myoelectric sensors, or a combination of two or more. Thesensors 1004 can send measured raw data to processor 1010, and processor1010 can perform noise cancellation to determine a signal correspondingto the user's gesture or motion. For devices that include at least twoof optical sensing, inertial sensing, mechanical contact sensing, andmyoelectric sensing, processor 1010 can dynamically activate the sensorsbased on an application and calibration. In some examples, one or moreof the sensors can be activated, while other sensors can be deactivatedto conserve power. In some examples, processor 1010 can store the rawdata and/or processed information in a ROM 1018 or RAM 1022 forhistorical tracking or for future diagnostic purposes.

In some examples, the sensors can measure the signal and processor 1010can determine the user's gesture and/or motion. In some examples,determination of user gesture and/or motion need not be performed on thedevice itself. FIG. 11 illustrates an exemplary configuration in which adevice is connected to a host according to examples of the disclosure.Host 1110 can be any device external to device 1100 including, but notlimited to, any of the systems illustrated in FIGS. 1A-1C or a server.Device 1100 can be connected to host 1110 through communications link1120. Communications link 1120 can be any connection including, but notlimited to, a wireless connection and a wired connection. Exemplarywired connections include Universal Serial Bus (USB), FireWire,Thunderbolt, or any connection requiring a physical cable.

In operation, instead of determining a user gesture and/or motion on thedevice 1100 itself, device 1100 can send raw data 1130 measured from thesensors over communications link 1120 to host 1110. Host 1110 canreceive raw data 1130, and host 1110 can process the light information.Processing the light information can include canceling or reducing anynoise due to artifacts and determining the user gesture and/or motion.Host 1110 can include algorithms or calibration procedures to accountfor differences in a user's characteristics or performance affecting thesensor signal. Additionally, host 1110 can include storage or memory fortracking a user gesture and motion history for diagnostic purposes. Host1110 can send the processed result 1140 or related information back todevice 1100. Based on the processed result 1140, device 1100 can notifythe user or adjust its operation accordingly. By offloading theprocessing and/or storage of the light information, device 1100 canconserve space and power, enabling device 1100 to remain small andportable, as space that could otherwise be required for processing logiccan be freed up on the device.

In some examples, a portable electronic device is disclosed. Theportable electronic device may comprise: one or more light emitterscapable of emitting light at a user's body part; one or more opticalsensors capable of detecting a first reflectance of the emitted light,wherein the first reflectance is associated with movement of one or moretendons located in the body part; and logic capable of determining agesture from the first reflectance and further capable of associating acommand with the determined gesture. Additionally or alternatively toone or more examples disclosed above, in other examples, the devicefurther comprises a strap attached to the device, wherein at least oneof the one or more optical sensors and at least one of the one or morelight emitters are located on or in the strap. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the at least one of the one or more optical sensors located onor in the strap is capable of detecting a second reflectance of emittedlight from the at least one or more light emitters located on or in thestrap, and the logic is further capable of determining the gesture fromthe first and second reflectance. Additionally or alternatively to oneor more examples disclosed above, in other examples, the devicecomprises at least two light emitters and at least two optical sensors,wherein the at least two light emitters and the at least two opticalsensors emit and detect light at different wavelengths. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the different wavelengths are selected from a group comprisinginfrared, blue, and green wavelengths. Additionally or alternatively toone or more examples disclosed above, in other examples, the opticalsensors are multi-functional sensors capable of detecting aphotoplethysmography signal. Additionally or alternatively to one ormore examples disclosed above, in other examples, the device furthercomprises at least one of an inertial sensor, a mechanical contactsensor, and a myoelectric sensor.

In some examples, a portable electronic device is disclosed. Theportable electronic device may comprise: a strap attached to the device,wherein the strap comprises a first band; and logic capable of measuringa change in one or more characteristics associated with movement of thefirst band in response to movement of one or more tendons located in auser's body part, determining a gesture based on the change in the oneor more characteristics, and associating a command with the gesture.Additionally or alternatively to one or more examples disclosed above,in other examples, the first band comprises a plurality of regions, theplurality of regions capable of stretching or compressing in response tothe movement of the one or more tendons, and wherein the one or morecharacteristics is a resistance due to a change in stretch orcompression in at least one of the plurality of regions. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the strap further comprises a second band, the first bandcomprises a plurality of optical features, and the second band comprisesone or more light emitters capable of emitting light at the opticalfeatures, and one or more light sensors capable of detecting areflectance of the emitted light, and wherein the one or morecharacteristics is the detected reflectance. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the device further comprises at least one of an opticalsensor, an inertial sensor, and a myoelectric sensor.

In some examples, a portable electronic device is disclosed. Theportable electronic device may comprise: one or more electrodes capableof detecting a change in capacitance associated with movement of one ormore tendons located in a user's body part; and logic capable ofdetermining a gesture based on the movement and further capable ofassociating a command with the determined gesture. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the one or more electrodes are located in or on a strapattached to the device. Additionally or alternatively to one or moreexamples disclosed above, in other examples, the device furthercomprises at least one of an optical sensor, an inertial sensor, and amechanical contact sensor. Additionally or alternatively to one or moreexamples disclosed above, in other examples, the device furthercomprises a transceiver capable of receiving a second gesture ormovement information from a second device, the second device capable ofdetecting the second gesture or movement associated with one or moretendons located on a second body part, wherein the logic is furthercapable of associating the command with the second gesture or movement.

In some examples, of method of determining a gesture is disclosed. Themethod may comprise: detecting a signal, wherein the signal is areflectance, change in capacitance, or change in resistance associatedwith movement of one or more tendons located in the body part;determining the gesture from the signal; and associating a command withthe determined gesture. Additionally or alternatively to one or moreexamples disclosed above, in other examples, the signal is a reflectanceof light, the reflectance of light being a reflectance profile generatedfrom a plurality of optical sensors detecting light at differentwavelengths. Additionally or alternatively to one or more examplesdisclosed above, in other examples, the plurality of optical sensors arecapable of detecting a photoplethysmography signal. Additionally oralternatively to one or more examples disclosed above, in otherexamples, the signal is a change in resistance generated from themovement of the one or more tendons causing a change in stretch orcompression in a strap. Additionally or alternatively to one or moreexamples disclosed above, in other examples, the determining the gestureincludes receiving a second gesture or movement information from anotherdevice, and the associated command is further based on the secondgesture.

Although the disclosed examples have been fully described with referenceto the accompanying drawings, it is to be noted that various changes andmodifications will become apparent to those skilled in the art. Suchchanges and modifications are to be understood as being included withinthe scope of the disclosed examples as defined by the appended claims.

What is claimed is:
 1. A portable electronic device comprising: a touchscreen included in a housing of the device; a plurality of first lightemitters included in the housing of the device, each first light emitterconfigured to emit first light at a user's body part; a plurality offirst optical sensors included in the housing of the device, each firstoptical sensor optically coupled to one of the plurality of first lightemitters to form one of a plurality of first emitter-sensor pairs andconfigured to detect a first reflectance, the first reflectance being areflectance of the emitted first light, wherein each firstemitter-sensor pair is associated with a different tendon located in thebody part than others of the plurality of first emitter-sensor pairs; aplurality of second light emitters included in a strap of the device,each second light emitter configured to emit second light at the user'sbody part, wherein the strap is separate and distinct from the housingof the device, the strap is attached to the housing of the device; and aplurality of second optical sensors included in the strap of the device,each second optical sensor optically coupled to one of the plurality ofsecond light emitters to form one of a plurality of secondemitter-sensor pairs and configured to detect a second reflectance, thesecond reflectance being a reflectance of the emitted second light,wherein each second emitter-sensor pair is associated with a tendonlocated in the body part; and logic included in the housing of thedevice configured to: determine a gesture from the first and secondreflectances detected by the first and second optical sensors,respectively, and associate a command with the determined gesture.
 2. Amethod of determining a gesture comprising: detecting one or more touchinputs using a touch screen located in a housing of a device; emitting aplurality of first light at a user's body part using a plurality offirst light emitters located in the housing of the device, each firstlight emitted at a different tendon located in the user's body part thanothers of the emitted first light; for each emitted first light from theplurality of first light emitters, detecting a change in firstreflectance using a plurality of first optical sensors located in thehousing of the device, the change in first reflectance indicative ofmovement of the associated tendon; emitting a plurality of second lightat a plurality of tendons included in the user's body part using aplurality of second light emitters located in a strap of the device, thestrap separate and distinct from the housing of the device, the strapattached to the housing of the device; for each emitted second lightfrom the plurality of second light emitters, detecting a change insecond reflectance using a plurality of second optical sensors locatedin the strap of the device; determining a gesture from the detectedchanges in first and second reflectances using logic located in thehousing of the device; and associating a command with the determinedgesture.
 3. The device of claim 1, wherein at least two of the pluralityof first light emitters and at least two of the plurality of firstoptical sensors emit and detect light at different wavelengths.
 4. Thedevice of claim 3, wherein the different wavelengths are selected from agroup comprising infrared, blue, and green wavelengths.
 5. The device ofclaim 1, wherein the first optical sensors are multi-functional sensorsconfigured to detect a photoplethysmography signal.
 6. The device ofclaim 1, further comprising an inertial sensor configured to determinetransitional and rotational motion, the inertial sensor including anaccelerator, and a gyroscope, wherein the gesture is further determinedfrom the transitional and rotational motion.
 7. The device of claim 1,further comprising a mechanical contact sensor configured to detect andconform to movements associated with the user's body part, and whereinthe gesture is further determined from the detected movements.
 8. Thedevice of claim 1, further comprising a myoelectric sensor configured todetect electrical signals associated with the user's body part, andwherein the gesture is further determined from the detected electricalsignals.
 9. A portable electronic device comprising: a touch screenincluded in a housing of the device; a strap attached to the housing ofthe device, wherein the strap comprises an inner band and an outer band,the inner band includes: a plurality of optical features, and the outerband includes: a light emitter configured to emit light at the pluralityof optical features, and an optical sensor configured to detect areflectance of the emitted light, wherein the reflectance is associatedwith movement of at least one tendon located in a user's body part; andlogic configured to: measure a change in characteristics associated withmovement of the inner band in response to movement of the at least onetendon located in the user's body part, determine a gesture based on thechange in the characteristics, and associate a command with the gesture.10. The device of claim 9, wherein the inner band comprises a pluralityof regions, the plurality of regions configured to stretch or compressin response to the movement of the at least one tendon, wherein thecharacteristics is a resistance due to a change in the stretch orcompression in at least one of the plurality of regions.
 11. The deviceof claim 9, further comprising an inertial sensor configured todetermine transitional and rotational motion, the inertial sensorincluding an accelerator and a gyroscope, wherein the gesture is furtherdetermined from the transitional and rotational motion.
 12. The deviceof claim 9, wherein the inner band is made of a flexible material, andthe outer band is made of a rigid material.
 13. The device of claim 9,wherein the plurality of optical features form a pattern from spatiallyseparated optical features, and the pattern is disposed around acircumference of the strap.
 14. The device of claim 9, furthercomprising a myoelectric sensor configured to detect electrical signalsassociated with the user's body part, and wherein the gesture is furtherdetermined from the detected electrical signals.
 15. The method of claim2, further comprising: recording the association of the command with thedetermined gesture.
 16. The method of claim 2, wherein each emittedfirst and second light includes a different wavelength than others ofthe emitted first and second light.
 17. The method of claim 2 furthercomprising: measuring a photoplethysmography signal using the pluralityof first light emitters and the plurality of first optical sensors. 18.The method of claim 2, wherein the determining the gesture includesreceiving a second gesture or movement information from another device,and the associated command is further based on the second gesture. 19.The method of claim 2, further comprising: determining translational androtational motion using an accelerometer and a gyroscope, whereindetermining the gesture further includes determining the gesture fromthe determined translational and rotational motion.